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The characterization of functional network structures among multiple neurons is essential 
to understanding neural information processing. Information geometry (IG), a theory 
developed for investigating a space of probability distributions has recently been applied 
to spike-train analysis and has provided robust estimations of neural interactions. Although 
neural firing in the equilibrium state is often assumed in these studies, in reality, neural 
activity is non-stationary. The brain exhibits various oscillations depending on cognitive 
demands or when an animal is asleep. Therefore, the investigation of the IG measures 
during oscillatory network states is important for testing how the IG method can be 
applied to real neural data. Using model networks of binary neurons or more realistic 
spiking neurons, we studied how the single- and pairwise-IG measures were influenced 
by oscillatory neural activity. Two general oscillatory mechanisms, externally driven 
oscillations and internally induced oscillations, were considered. In both mechanisms, we 
found that the single-IG measure was linearly related to the magnitude of the external 
input, and that the pairwise-IG measure was linearly related to the sum of connection 
strengths between two neurons. We also observed that the pairwise-IG measure was not 
dependent on the oscillation frequency. These results are consistent with the previous 
findings that were obtained under the equilibrium conditions. Therefore, we demonstrate 
that the IG method provides useful insights into neural interactions under the oscillatory 
condition that can often be observed in the real brain. 
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INTRODUCTION 

The dynamics of neural interactions have been conjectured to 
play an important role in neural information processing. One way 
to investigate the neural interactions is to record multi-neuronal 
firing activity from a freely behaving animal, and analyze the cor- 
relations between individual units. In past decades, electrophys- 
iological studies have significantly been advanced by the use of 
multi-electrode recording techniques (Wilson and McNaughton, 
1993; Chapin et al., 1999; Kudrimoti et al., 1999; Laubach et al., 
2000; Hoffman and McNaughton, 2002; Buzsaki, 2004; Tatsuno 
et al, 2006; Euston et al, 2007; Davidson et al, 2009; Peyrache 
et al., 2009b; Dragoi and Tonegawa, 2011, 2013). In order to ana- 
lyze such high-dimensional multi-neuronal datasets, a number of 
statistical methods have also been developed (Gerstein and Perkel, 
1969; Abeles and Gerstein, 1988; Aertsen et al., 1989; Zhang et al, 
1998; Panzeri and Schultz, 2001; Grun et al, 2002a,b; Brown et al., 
2004; Fellous et al, 2004; Czanner et al, 2005; Shimazaki and 
Shinomoto, 2007; Amari, 2009; Gilestro et al., 2009; Peyrache 
et al., 2009a; Shimokawa and Shinomoto, 2009; Lopes-Dos- 
Santos et al., 2011). Recently, a method based on information 
geometry (IG) has been applied to the analysis of neural data 
(Amari and Nagaoka, 2000; Amari, 2001; Nakahara and Amari, 
2002; Amari et al., 2003; Tatsuno and Okada, 2004; Eleuteri et al, 
2005; Ikeda, 2005; Miura et al, 2006; Nakahara et al., 2006; 



Gilestro et al, 2009; Tatsuno et al, 2009; Ince et al, 2010; Lovette 
et al., 2011; Nie and Tatsuno, 2012). It has been demonstrated 
that IG provides a powerful statistical tool for analyzing spiking 
data. Some of the advantages of IG approach include the orthogo- 
nal decomposition of neural interactions (Amari, 2001; Nakahara 
and Amari, 2002), and its direct relationship to underlying con- 
nections (Tatsuno and Okada, 2004; Tatsuno et al., 2009; Nie and 
Tatsuno, 2012); the single-IG measure is related to the amount 
of external inputs and the pairwise-IG measure is related to the 
amount of direct neural interactions between two neurons. 

These IG properties were often investigated under the assump- 
tion that the network is in an equilibrium state. However, in the 
brain, the equilibrium assumption does not hold true. Instead, 
the brain undergoes a variety of non-equilibrium states such 
as oscillations. For example, the slow- wave oscillation (~1 Hz) 
was discovered during non-REM sleep (Steriade et al., 1993; 
Crunelli and Hughes, 2010), and evidence suggests that it plays 
an important role in memory consolidation (Huber et al., 2004; 
Stickgold, 2005; Diekelmann and Born, 2010). The theta (6- 
10 Hz) rhythm is a prominent coherent oscillation observed 
in the hippocampus, and its surrounding area during rat spa- 
tial navigation (Vanderwolf, 1969; Bland, 1986; Buzsaki, 2002). 
The theta rhythm has also been observed in various human 
neocortical areas during the delay period of working memory 
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tasks (Raghavachari et al., 2001; Meltzer et al., 2008). The beta 
(15-30 Hz) oscillation is conjectured to play a key role in action 
preparation and inhibitory control in the motor system (Baker 
et al, 1997). The gamma (30-80 Hz) oscillation has been shown 
to play a role in the integration of sensory information (Gray 
et al., 1989; Singer and Gray, 1995). The fast hippocampal sharp 
wave ripples (100-200 Hz) were also reported during an animal's 
awake immobility and slow-wave sleep (Buzsaki et al., 1992). 
Therefore, it is important to investigate if the IG measures can 
be applied to neural data under oscillatory conditions. 

In this study, we investigated how the single- and pairwise- 
IG measures are influenced by a network oscillation. Under an 
equilibrium assumption, previous studies have shown that the 
single- and pairwise-IG measures provide a robust estimation of 
the magnitude of external input and direct neural interactions 
(Tatsuno et al., 2009; Nie and Tatsuno, 2012). We also focused on 
these IG measures in this study because the external inputs and 
intrinsic neural interactions are the two main factors for char- 
acterizing network dynamics. For the oscillation mechanisms, we 
have considered two representative cases; one is an external driven 
oscillation where a network is influenced by external oscillatory 
inputs. The other is an internally induced oscillation where inter- 
actions between excitatory and inhibitory neuron populations 
produce an oscillation. By computer simulations using simple 
binary model neurons or more biologically plausible spiking neu- 
rons, we investigated whether the properties of the IG measures 
that were established with the equilibrium condition still hold 
true under oscillatory network states. 

In section Methods, we briefly introduce an information- 
geometric analysis of neural spikes. In section Results, we 
describe the model and network structure used in the numeri- 
cal simulation. In section Discussion, the simulation results for 
both externally driven and internally induced oscillations are 
described in detail. In section Acknowledgments, we summarize 
our findings and discuss future directions of research on this 
topic. 

METHODS 

INFORMATION-GEOMETRIC METHOD 

We briefly introduce the information-geometric method for spik- 
ing data analysis (for details see Amari and Nagaoka, 2000). 
Generally, in an N-neuron system, the state of i-th(i = 1, . . , , N) 
neuron is represented by a binary random variable x\, where 
Xi = 1 or 0 representing neuronal firing or silence, respectively. 
The joint probability distribution of the N-neuron system can be 
described by a fully expanded N-th order log-linear model (LLM) 



log£* 



and the number of neurons in the system. We use 9 , 0 j^' ^ , 

and Qj^.l.^ to describe the single-IG measure, the pairwise-IG 
measure and the m-neuron IG measure with the N-th order LLM 
for aN-neuron system, respectively, (Nie and Tatsuno, 2012). The 
joint probability of N neurons is calculated by 



(2) 



where c x 



is the count of events (Xi = x\ , X 2 



X2, . . . , = xn) that occur. 

However, in reality, it is difficult to calculate the statistical 
information from all neurons in a large network. Therefore, the 
partially expanded LLM is often used for the estimation of neu- 
ronal interactions. The partially expanded k-th order LLM in an 
N-neuron network is given by 



logpx 



AO 



(k, N) 



(3) 



where9=^ N) ,e (fc - N) 



a(k, N) 
ij ' • • • ' D 12,- k 

and normalization factor are given as follows 



The first few terms of 0 



a (k, N) , Pxi = 0, - , xi = 1, - , x k = 0,*-* 

0; = lOg , 

yx\ = 0,-" ,Xk = 0,* •■■ * 



e* N) = io g 



px\ = 0,..., %{ = 1,..., Xi = 1,— ,Xfc = 0,* ■■■ * 
Px\ = (),■■■ , Xk = 0,*---* 

Px\ = 0,..., Xj = 1,..., Xj = 0,..., Xk = 0, * ••• * 
Px\ = 0,..., Xj = 0,..., Xj = 1,..., Xk = 0, *•••* 



(4) 



(k, N) 



Togp Xl =o,..., x k = o, 



x k = 0, * ■■■ * 5 



where represents the marginalization over the (N — k) 

neurons. 

The single-IG measure O^' N ' and the pairwise-IG measure 
0« ' ^ are the two main focuses in this study because 0^ ' N ' is 

related to the amount of external inputs and 9» ' N ' is related 
to the amount of direct neural interactions between two neu- 
rons (Tatsuno and Okada, 2004; Tatsuno et al, 2009; Nie and 
Tatsuno, 2012). Using a network of simple binary neurons, and 
the assumption of an equilibrium state, the previous study has 
shown that the single-IG measure oj 2 ' ^ and the pairwise-IG 
measure 9^ 2 ' with the 2nd-order LLM are related to the net- 
work parameters as 



+ Q { n;.^x 1 x 2 ---x N -MQ) (N ' N \ (1) 



where 0- ; .1^/(1 < m < N) represents the m-neuron interaction 



and ^(e)< N - w > with 6 = (0f' N \ ef' N) , . . . 
malization constant such that X!P*i*2 "' x N 
the second superscripts in B^.'..^ represent the order of the LLM 



,(N, N) 
3 12,--N 

1. The first and 



i(2, AO 



a 2h t + O 



N 



f N U(j v +Jji) + o KN 



(5) 



where /z, represents the magnitude of constant external input to 
a neuron i, and h (/«) is the connection weight from a neuron j 
to i (from a neuron ; to j), respectively, (Tatsuno et al, 2009). If a 
network receives correlated inputs, the relationship for 0?' N ^ in 
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Equation 5 does not hold true anymore. However, we have also 
shown that 9| ' N ' with the higher k-th order LLM provides a bet- 
ter estimation of neural interactions (Nie and Tatsuno, 2012). For 
example, 6* 4 ' ^ was shown to have the relationship 



3 (4, N) 



(6) 



within approximately a 10% error if the number of neurons N 
is ~ 10 3 or larger; a typical size of network in a cortical column 
(Urban et al, 2001). We have also confirmed that the relation- 
ship 6 (4 ' N) ct2/z,- holds true within approximately a 10% error 
(unpublished data). 

These properties could be useful for the field of neuroscience 
because the IG measures can estimate the changes of underlying 
network parameters (?z; and h) separately, while other correla- 
tion measures have not yet been shown to have such a property 
(Amari, 2009). However, these results were derived under the 
equilibrium limit, and little is known if the similar relationship 
holds under the oscillatory condition. 

NEURON MODEL AND NETWORK STRUCTURE 
Neuron model 

We investigated the influence of oscillations using a network of 
simple binary neurons with stochastic dynamics (Ginzburg and 
Sompolinsky, 1994) and biologically plausible spiking neurons 
(Izhikevich, 2003). Using simple binary model neurons, we first 
investigated whether the property of the IG measures that were 
shown under the equilibrium condition also held true for the 
oscillatory condition. We then extended our investigation to more 
realistic spiking neurons. 

For a binary model neuron, the transition between the binary 
states is given by the transition rate w as 



w(xj = 0 — >■ Xi = 1) 



w(Xj 



0) 



to 

1 - 



W(Xi 

w(x i 



0) 
1) 



to 

1 - w(0 • 
1 - w(l ■ 



(7) 



1), 
0), 



where To is a microscopic characteristic time and tz, represents the 
total input to the z'-th neuron. 



1 + tanh(tz, — m) 



(8) 



is the sigmoidal function in the bounded interval [0, 1 ] where m is 
a parameter controlling the firing probability of a model neuron. 

For a biologically more plausible neuron model, we adopted 
the Izhikevich model because it is known to be computation- 
ally efficient and biologically plausible (Izhikevich, 2003). The 
Izhikevich model reduced the complex dynamics of the Hodgkin- 
Huxley (HH) neuronal models to two coupled differential equa- 
tions as 



Here the variable V; represents the membrane potential of neu- 
ron z, and 17; represents a membrane recovery variable which 
correlates with the activation of K + ionic currents and inacti- 
vation of Na + [for detail see (Izhikevich, 2003)]. 17,- and V, are 
reset after a spike: if V, > 30 mV, then V,- <— q, U, <— 17; + d\. I, 
represents a total input to neuron i; a,, bi, c,, d, are dimension- 
less adjustable parameters which are usually taken as (a,, bi) = 
(0.02, 0.2) and (c„ d,) = (-65, 8) + (15, -6)r 2 for excitatory 
neurons, (a,-, b { ) = (0.02, 0.25) + (0.08, -0.05r») and(c;, d,) = 
(—65, 2) for inhibitory neurons, r; is a uniformly distributed 
random variable on the interval [0, 1] (Izhikevich, 2003). 

Network structure 

We considered two mechanisms for generating oscillatory net- 
work states; one is the oscillation driven by external inputs 
(Figure 1A), and the other is the oscillation induced by the 
intrinsic interaction between excitatory and inhibitory neuron 
populations (Figure IB). The former mechanism can be a model 
for hippocampal theta oscillation in which the projections from 
the medial septum to the hippocampus play a central role (Dragoi 
et al., 1999). The latter structure where excitatory and inhibitory 
neuron pools interact is widely observed in cortical areas (Buzsaki 
and Wang, 2012). It can be a model for cortical oscillations (such 
as in a gamma-range) that rely on the interplay between excitatory 
and inhibitory neuron pools. 

In the first scenario (externally driven oscillation, Figure 1A), a 
sinusoidal external input h,{t) = ho sin(cof + op,) for the z'-th neu- 
ron was used to generate oscillatory states in a network, where 
ho, co, and upvarphi, represent the amplitude, angular speed, 
and phase of sinusoidal signals, respectively. Note that ho and 
co are common to all neurons, but (p,- can be different for indi- 
vidual neurons. The explicit expression of an input signal allows 
one to produce different network oscillations systematically. For 
the binary neuron model, the total input to the z'-th neuron is 
written as, 

= J2 ViW + W0- (10) 



where h represents a connection weight from the j-th neuron to 
the z'-th neuron. The neuronal state %,(f) was then updated follow- 
ing the transition rate w in Equation 7. Note that model neurons 
are identical, whether they are excitatory or inhibitory. 

For the Izhikevich model in the first scenario, we considered a 
population of excitatory neurons. Although it has been demon- 
strated that a network of excitatory neurons can synchronize, a 
network of Izhikevich neurons that were connected in this par- 
ticular way cannot produce an intrinsic oscillation (Mirollo and 
Strogatz, 1990; Hansel et al., 1995). This allows us to investi- 
gate the relationship between the IG measures and an externally 
driven oscillation in a more realistic setting. The total input to an 
Izhikevich neuron i is given by, 



If(t) 



^/fsfCO + MO, 



(11) 



dV, 
dt 



0.04Vf + 5V, + 140 - U, + 7„ 



dU, 
dt 



ii(b,V t — Ui). where J EE represent positive weights between excitatory neurons 
(9) and = 8(f — rj) is the delta function representing the existence 
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h t — /i 0 sin(dit + (pi) 



FIGURE 1 | A schematic of two mechanisms for generating network 
oscillations. (A) Oscillation is generated by an external oscillatory input 
(externally driven oscillation). A neuron / in the network of W neurons 
with recurrent connections Jjj receives a sinusoidal external input 
h, = ho sin(a>f + ip,-) where ho, w, and ip,- represent the amplitude, 
angular speed, and phase of the sinusoidal input, respectively. (B) 
Oscillation is generated by the interaction between excitatory and 



inhibitory neuron pools (internally induced oscillation). Excitatory neurons 
are connected by positive connections Jee, inhibitory neurons are 
connected by negative connections J;j, inhibitory neurons receive positive 
connections J/e from excitatory neurons, and excitatory neurons receive 
negative connections Jei from inhibitory neurons. In addition, excitatory 
and inhibitory neurons receive external constant input Hf and HL 
respectively. 



of a spike emitted from an excitatory neuron j at time f|. The 
neuronal state was then updated by Equation 9 and the associ- 
ated reset dynamics. In the numerical simulation, we used a small 
bin width of 1 ms so that it would contain no more than one 
spike. 

Figures 2A-C show example spike trains and multi-unit 
activity of binary model neurons driven by external sinu- 
soidal inputs of 1, 6, and 100 Hz oscillations, respectively. 
Izhikevich neurons also exhibited very similar activity (data 
not shown). It can be clearly seen that neural activity is 
entrained to external input. Using these two models, we inves- 
tigated how the IG measures are affected by externally driven 
oscillations. 

In the second scenario (internally induced oscillation, 
Figure IB), interaction between excitatory and inhibitory neuron 
pools generates an oscillation. For the binary neuron model, the 
total input to the z'-th excitatory neuron and the k-th inhibitory 
neuron are written as, 

«? w = E#) + £f *f<*> + R f • 

«*« = £ W> + £4^(0+^4 w 

i i 

where J u , J IE , and J E! represent negative weights between 
inhibitory neurons, positive weights from excitatory neurons 
to inhibitory neurons, and negative weights from inhibitory 
neurons to excitatory neurons, respectively. The excitatory 
and inhibitory neurons receive constant external inputs Hf 
and H 1 , and maintain sustained oscillatory activity. The neu- 
ronal state was updated following the transition rate w in 
Equation 7. 

For the Izhikevich model in the second scenario, a similar rela- 
tionship exists for the total inputs for the i-th excitatory neuron 



and the fc-th inhibitory neuron, 

if® = £/f *f « + £ 7 f *f (f) + H >> 

The neuronal state was then updated following Equation 9 and 
the associated reset dynamics. Figures 2D,E provide examples of 
spike trains and multi-unit activity of Izhikevich neurons that 
exhibited ~6 and 40 Hz oscillations, respectively. Binary neurons 
also exhibited a very similar activity (data not shown). Neural 
activity was synchronized, but the degree of entrainment was 
weaker than the externally driven mechanisms. Using these two 
models, we investigated how the IG measures were influenced by 
the internally induced oscillation. 

RESULTS 

EXTERNALLY DRIVEN OSCILLATION 

We investigated the relationship between the IG measures, 9^ 4 ' N ^ 
and 9^ 4 ' , and the connection weights, + Jj t ), using a net- 
work of 1000 binary neurons and 1000 Izhikevich neurons. 
We focused on the IG measures with 4th-order LLM because 
they have been shown to estimate connection weights (Nie and 
Tatsuno, 2012) and external inputs (unpublished data) within 
a 10% error under an equilibrium assumption. In the simula- 
tion, we kept the amplitude of external input at a value such that 
the overall network firing probability is relatively low (p Xj ~ 0.1). 
Connection weights were set to the order of 1 /N to prevent sat- 
uration of neuronal activity. For a binary neuron model, we used 
h = 1 /N + £jj where Sy is a random variable from a normal dis- 
tribution N(0, 1 /N) with a mean of 0 and the standard deviation 
of 1 /*JN. For the Izhikevich model, we restricted the simulations 
to a pool of only excitatory neurons to ensure that no inter- 
nally induced oscillation occurred. The connection weights were 



Frontiers in Neural Circuits 



www.frontiersin.org 



February 2014 | Volume 8 | Article 11 | 4 



Nie et al. 



IG estimation of neural interactions 



0.2r 



0.1 



^^^^ 




200 



400 600 
ms 



800 



1000 



0.2 



0.1 



AAAAAA 



100 



200 



400 600 
ms 



800 



1000 




0 
100 



5 50 







SB 


■y,,-.,,;:.:- 


■ 














■ ■ 








: ■ ■ 





200 



400 600 
ms 



800 



1000 



0.1 



-Q 

o 



0.05 



0.2 



o 
a 



0.1 




AAA 




400 600 
ms 



1000 



1000 



500 





LlLikiiliiiiA 




lilffllllfi 





200 



400 600 
ms 



800 



1000 



FIGURE 2 | Average firing probability and raster plot of 
representative oscillatory activity. (A) Average firing probability of 
1000 binary neurons (top panel) and a raster plot of 100 randomly 
sampled neurons (bottom panel) over 1000 ms under the influence of 
an external sinusoidal input of 1 Hz (slow oscillation) are shown. (B) 
Average firing probability of 1000 binary neurons (top panel) and a 
raster plot of 100 randomly sampled neurons (bottom panel) for an 
external sinusoidal input of 6 Hz (theta oscillation) are shown. (C) 
Average firing probability of 1000 binary neurons (top panel) and a 



raster plot of 100 randomly sampled neurons (bottom panel) for an 
external sinusoidal input of 100 Hz (ripple oscillation) are shown. (D) 
1000 ms of average firing probability of 1250 Izhikevich neurons (top 
panel) and a raster plot (bottom panel) with approximately a 6-Hz 
oscillation are shown. In the top panel, spikes from an excitatory 
neuron and an inhibitory neuron are represented by a black dot and a 
gray dot, respectively. (E) 1000 ms of average firing probability of 1250 
Izhikevich neurons (top panel) and a raster plot (bottom panel) with 
approximately a 40-Hz oscillation are shown. 



assigned as J^ E = l/N + e'ij where e'y is a random variable fol- 
lowing uniform distribution [7(0, l/N) within the interval of 
[0, l/N]. e, (4 - N) and O^ 4 ' N) were calculated by 10 6 updates of 
the network. With the time resolution of 1 ms, the simulation 
corresponds to ~15min of recordings. To obtain the mean and 
variances of the IG measures, we performed 100 independent 
simulations. Error bars in the figure represent the standard error 
of mean (SEM). 

We investigated the oscillation frequencies that have often been 
observed in the brain; slow oscillation (~1 Hz), theta oscillation 



(6-10 Hz), and ripple oscillation (100-200 Hz). The left column 
of (Figures 3A,D,G) shows the results for the slow oscillation. 
The multi-unit activity of the binary neurons exhibits a slow 
oscillation of the frequency of external input (Figure 3A) and 
the neurons were entrained to this frequency (Figure 2A). The 
spiking activity of Izhikevich neurons also showed almost iden- 
tical activity (data not shown). To investigate how S^ 4, N ^ and 
6j 4 ' N ^ are related to the change of connection weights, we sys- 
tematically modified the sum of connection weights between two 
neurons (1 and 2) from —9/N to 9/N. Due to the randomness 
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FIGURE 3 | Relationship between the IG measures and the sum of 
connection weights for an externally driven oscillation. (A) Average firing 
probability of 1000 binary neurons with a 1-Hz oscillatory modulation is 
shown. (B) Average firing probability of 1000 binary neurons with a 6-Hz 
oscillatory modulation is shown. (C) Average firing probability of 1000 binary 
neurons with a 100-Hz oscillatory modulation is shown. (D) Relationship 
between the pairwise IG measure (6& 1000) ) and the sum of connection 
weights (J12 + J21) under a 1-Hz oscillation. Black and gray lines represent 
the simulation results by binary neurons and Izhikevich neurons, respectively. 
(E) Relationship between the pairwise IG measure (8*1' 1000 >) and the sum of 



connection weights (J 12 + J21 ) under a 6-Hz oscillation. (F) Relationship 
between the pairwise IG measure (flit' 1000) ) and the sum of connection 
weights Uu + J21 ) under a 100-Hz oscillation. (G) Relationship between the 
single IG measure (8* 4, 1000) ) a nd the sum of connection weights (J 12 + J21) 
under a 1-Hz oscillation. Black and gray lines represent the simulation results 
by binary neurons and Izhikevich neurons, respectively. (H) Relationship 
between the single IG measure (8 ( , 4, 1000) ) and the sum of connection 
weights (J12 + J21) under a 6-Hz oscillation. (I) Relationship between the 
single IG measure (8* 4, 1000 >) a nd the sum of connection weights (J 12 + J21) 
under a 100-Hz oscillation. 



of the connectivity, focusing the neurons (1 and 2) did not 

affect the generality. We found that N) was linearly related 
to the sum of the connection weights, and that the values of 
®12 N ^ f° r both the binary and Izhikevich models were very close 
(Figure 3D, black line for a binary model and gray line for the 
Izhikevich model). On the other hand, 6j 4 ' N ^ and Q^' were 
independent from the change of synaptic weights (Figure 3G). 
These results are consistent with the previous findings under 
the equilibrium assumption; showing that IG measures can also 
provide useful insights in conditions where the network oscil- 
lates. The middle and right columns of Figure 3 show the results 



for theta oscillations (Figures 3B,E,H) and ripple oscillations 
(Figures 3C,F,I), respectively. We found that the relationship 
between the IG measures and connection weights was robust 
against a different frequency of external inputs. This confirmed 
that the IG measures can also provide useful information for 
externally driven theta and ripple oscillations. 

To further investigate if the robust property of the IG measures 
for slow, theta, and ripple oscillations holds true for other fre- 
quencies, we varied the frequency over 1-200 Hz, the range that 
can be typically observed in the brain. We set (Jn + hi) = 2/. 
Figure 4 shows that 0^' and Q* 4, did not depend on 
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FIGURE 4 | Dependency of the IG measures for an oscillation frequency 
for an external drive oscillation. The values of the single-IG measure 
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neurons. (C) The relationship between 8**' 1000) and the oscillation frequency 
by binary model neurons. (D) The relationship between 8* 4 ' 1000) and the 
oscillation frequency by Izhikevich neurons. 



oscillation frequencies (Figures 4A,C for a binary model, and 
Figures 4B,D for Izhikevich model). The results confirmed that 
the IG measures would be useful for neural data analysis when 
the brain exhibits a variety of oscillations depending on cognitive 
demands and the sleep stages. 

The previous analyses (Figures 3, 4) were performed under the 
zero relative phase difference between two neurons i and;', namely 
Scpy = | cp, — (p ; | =0. This corresponds to the synchronous neural 
firings that were depicted in Figures 2A-C. Neurons can, how- 
ever, exhibit phase differences. For instance, sequential neural 
activity was observed in the natural and anesthetized brain states 
(Lee and Wilson, 2002; Euston et al, 2007; Luczak et al, 2007; 
Bermudez Contreras et al., 2013). Therefore, we calculated IG 
measures with phase differences. Figure 5 shows the results of 
the 6-Hz simulations in which the phase difference between sinu- 
soidal inputs to the neurons 1 and 2 was set to it /6 (Figures 5A,C) 
and Tt/2 (Figures 5B,D). The rest of the neuron pairs have ran- 
dom phases in the range of [0, lit]. Figures 5A,B show that 
0j2 100 °^ is linearly related to the sum of synaptic weights, sug- 
gesting that the relationship observed in zero phase difference 
condition also holds for the non-zero phase difference condition. 
Similarly, Figures 5C,D show that 9j 1000 ' does not depend on 
the connection weights, even when neurons fire with phase dif- 
ferences. By comparing these results with Figures 3E,H where 
there was no phase difference, we also found that phase differ- 
ence produced the shift of the actual values of IG measures. This 
suggests that if the phase relationship drastically changes between 



the two recording epochs, the values of the IG measures cannot 
be directly comparable. However, if their difference is not large or 
if phase difference can be estimated beforehand, we could use the 
information for adjusting the IG values. We also confirmed that 
these relationships held true for slow (1 Hz) and ripple (100 Hz) 
frequencies (data not shown). 

So far we have focused on the relationship between the IG 
measures and the connection weights. Another important param- 
eter is the magnitude of sinusoidal input ho- Therefore, we have 
analyzed how q\ 4 ' 1000 ' and Qy£ 1000 ' are related to ho- Figure 6 
shows the result when the external sinusoidal input has a fre- 
quency of 6 Hz (theta oscillation). We found that 6^' 1000) was 
nearly independent from the change of ho (Figure 6A), but 
9j 1000> was almost linearly related to it (Figure 6B). We also 
confirmed that almost identical relationship holds true for other 
frequencies such as slow oscillation and ripple oscillation if there 
is no phase difference. For non-zero phases between neurons, we 
observed that the IG values were shifted, like the case for the IG 
values and connection weights, but that the same linear and inde- 
pendent relationship in Figure 6 was sustained. The results under 
the oscillatory condition are consistent with the previous findings 
under the equilibrium condition; 9 ; - 4 ' N ^ was linearly related to the 
magnitude of the constant input and that Q^' N ' was almost inde- 
pendent from it (Nie and Tatsuno, 2012). The investigation here 
provides further evidence that 9- N ' is useful for the estimation 
of the magnitude of external input. 
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FIGURE 5 | Relationship between the IG measures and the sum of 
connection weights for non-zero phase differences. An external oscillation 
mechanism was used, and the oscillation frequency was set to 6 Hz. (A) 
Relationship between the pairwise IG measure ^S* 4 ,' 10C0) ) and the sum of 
connection weights (Ju + J21 ) for the phase difference of Sip n 2 = ir/6. Black 
and gray curves represent the simulation by binary neurons and Izhikevich 
neurons, respectively. (B) Relationship between the pairwise IG measure 



^9*2 1000) ^ and the sum of connection weights, (J 12 + J21 ) for the phase 
difference of Sip-]2 = it/2. (C) Relationship between the single-IG measure 
n}' 4 ' 1000) ^ and the sum of connection weights (J12 + J21 ) for the phase 
difference of Scp-12 = ji/6. (D) Relationship between the single-IG measure 
^g(4, 1000)^ gnc j ^ e sum o ^ connect i on weights (J-]2 + J21 ) for the phase 
difference of Scp-12 = n/2. 
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FIGURE 6 I Relationship between the IG measures and the amplitude of 
an external sinusoidal input for an externally driven oscillation. 

Oscillation frequency was set to 6 Hz and phase difference was 81^2 = 0. (A) 
Relationship between the pairwise-IG measure, S^ 4, 1000) , and the amplitude 



of a sinusoidal input, hg. Black and gray lines represent the simulation results 
by binary neurons and Izhikevich neurons, respectively. (B) Relationship 
between the single-IG measure, 8^ 4 wa0 \ and the amplitude of a sinusoidal 
input, /lo- 



in summary, we investigated how the IG measures were 
influenced by an externally driven oscillation. Using a simple 
binary neuron model, and a more realistic Izhikevich model, 
we found that 8^ 4 ' N ' had a linear relationship with the sum 
of the connection weights, and that it was almost independent 
from the magnitude of a sinusoidal input. In contrast, 6^ 4 ' N ^ was 
almost independent from the connection weights, but was linearly 



related to the magnitude of the sinusoidal input. These proper- 
ties were not affected by the frequency of the oscillations or the 
relative phase differences between neurons. 

INTERNALLY INDUCED OSCILLATION 

As another mechanism for generating an oscillatory network 
behavior, we also investigated the interactions between excitatory 
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and inhibitory neuron pools. We analyzed the network structure 
in Figure IB by simple binary model neurons and Izhikevich neu- 
rons. Unlike the first oscillation mechanism, where an oscillation 
frequency and phase differences could be explicitly controlled, it 
was not easy to generate an oscillation with desired parameters. 
However, we were able to generate two examples that were often 
observed in the brain. Figures 7A,B show multi-unit activity cor- 
responding to theta frequency (~8 Hz) and gamma frequency 
(~40 Hz), respectively. The same examples with a raster plot were 
also depicted in Figures 2D,E. To avoid saturation in neural activ- 
ity, we have set the connection weights to the order of 1 /TV. For 
a theta oscillation, we set the connection as J EE = J\ ■ W = 

5/i • 4, 7? = -h ■ e'ij and jf = -J 2 ■ e'.. where h = l/N e , h = 
l/N{, and e'a is a random variable following a uniform distri- 
bution 17(0, 1) within the interval [0,1]. For a gamma oscil- 
lation, we used J EE = h • e'.., } IE = 6/1 • e'.., /? = -J 2 • and 
J EI = — 2/ 2 - The stronger J IE was necessary to induce oscilla- 
tion (Adini et al., 1997). The external constant inputs to excitatory 
and inhibitory neurons were set as H E = 0.05 and H\ = 0.02, 
respectively. The simulation of 10 6 update was performed with 
N e = 1000 excitatory neurons and TV,- = 250 inhibitory neurons. 
The mean and variance was estimated using 100 independent 
simulations. 

To investigate the relationship between the IG measures and 
the sum of connection weights, without losing the general- 
ity, we modified (/ 12 + hi) between the neurons (1 and 2) 
in the range of [—9/, 9/]. Firstly, we focused on the connec- 
tions within the excitatory neuron population and within the 
inhibitory neuron population. In other words, both connections, 
Jn and hi, were positive for the range of (/ 12 + hi) > 0 and 
both were negative for (J u + hi) < 0. Figures 7C,E show the 
relationship between 9j 4 ' 125 °' and 9j 4 ' 1250 \ and the sum of con- 
nection strengths (/ 12 + /21) under the theta oscillation. The 
results clearly show that Oj 4 ,' 125 °' ) is linearly related to the sum 
of the connection weights and that 9j 1250) was independent 
from the modulation of the connection weights. Furthermore, 
the dependency of the IG measures on the connection weights 
was continuous in both positive and negative ranges. This sug- 
gests that IG measures can be applicable to both positive and 
negative connections. Figures 7D,F show results for gamma oscil- 
lation. Similar results were obtained for both 9j 2 125 °' ) and 

q(4, 1250) 

Secondly, we investigated the interaction between excitatory 
and inhibitory neurons. Namely, we selected the neuron 1 from 
the excitatory neuron pool and the neuron 2 from the inhibitory 
neuron pool. The sum of connection weights was modified 
from —9] to 9/. Figures 8A,B are the same with Figures 7A,B, 
showing the multi-unit activity for theta and gamma oscillation, 
respectively. Figures 8C,E show the relationship between the IG 
measures and the sum of the connection weights under the theta 
oscillation. Similarly, Figures 8D,F are for gamma oscillation. The 
results show that the linear dependency of 9 j 4 ' 1250 ' ) on the sum of 
the connection weights holds true for an excitatory and inhibitory 
neuron pair. We also found that 9[ 4 ' 1250 ' had almost no rela- 
tionship with the sum of connection weight. For the relationship 



between the IG measures and the magnitude of constant input H E 

and H\, we confirmed that 9^ 4 ' 1250 ' ) was linearly related to their 

magnitude, but 9j 4, 125 °' was independent from them (data not 
shown). 

In summary, for internally generated oscillations, we demon- 
strated that the relationship between the IG measures and the 
connection weights that were found under equilibrium assump- 
tion also held true. 

DISCUSSION 

Previous studies have shown that the IG measures provided use- 
ful information about network structures (Tatsuno and Okada, 
2004; Tatsuno et al, 2009; Nie and Tatsuno, 2012). Specifically, 
the single-IG measure 9| 4 ' ^ was related to the magnitude of 
external constant input, and the pairwise-IG measure 9?' N) was 
related to the sum of the connection strengths. Although these 
studies were conducted under the equilibrium assumption, the 
real neural signals exhibit various oscillations depending on cog- 
nitive demand of the task or the state of the brain. Therefore, we 
studied the relationship between the IG measures and the neural 
network parameters under oscillatory network states. 

We have considered two general oscillation mechanisms; one 
was the oscillation driven by external input, and the other was the 
oscillation induced internally due to interactions between exci- 
tatory and inhibitory neuron pools. Numerical simulation was 
performed by the network of a simple binary neuron model and 
the Izhikevich neuron model. The former model was used so as 
to compare the results with that of previous studies, and the latter 
was used to investigate the relationship with more realistic model 
neurons. 

For the external oscillation, our investigation showed that 
Q^p ™ was linearly related to the sum of the connection strengths, 

and that 9- 4, N%> was independent from it over a wide range of fre- 
quency from 1 to 200 Hz. We also showed that the relationship 
holds true when there are phase differences between neurons. In 
addition, we demonstrated that 9[ 4, ^ was almost linearly related 
to the magnitude of sinusoidal input, but that 9- 4 ' N ' was almost 
independent from it. For the internally induced oscillation, we 
have also confirmed that 9^ 4 ' N ' was linearly related to the sum of 

the connection strengths, and that 9* 4 ' N ~* was independent from 
it. We have also shown that the same relationship holds true for 
any neuron pairs (within excitatory population, within inhibitory 
population, and across excitatory and inhibitory populations). 

In summary, this study and previous studies have demon- 
strated that the IG measure provides useful information for 
analyzing neural circuits; not only for the equilibrium condition, 
but also for the oscillatory condition. The single-IG measure is 
useful for estimating the relative strength of external inputs. In 
addition, the single-IG measure is better than using the change in 
firing rate because the firing rate can be modulated both by the 
change in synaptic coupling strength and the magnitude of exter- 
nal inputs. Studies show that the appropriately selected single-IG 
measure is capable of estimating the external inputs with rela- 
tively small influence from synaptic interactions. Similarly, the 
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FIGURE 7 | Relationship between the IG measures and the sum of 
connection weights for an internally induced oscillation. Both 
connections, J12 and J21 . were positive for the range of (J 12 + J21 ) > 0 and 
they were negative for (J 12 + J21 ) < 0- (A) Average firing probability of 1250 
Izhikevich neurons with approximately a 6-Hz oscillatory oscillation is shown. 
(B) Average firing probability of 1250 Izhikevich neurons with approximately a 
40-Hz oscillatory oscillation is shown. (C) Relationship between the 
pairwise-IG measure (8^' 1000) ) and the sum of connection weights 



(J12 + J21 ) under a 6-Hz oscillation. Black and gray lines represent the 
simulation results by binary neurons and Izhikevich neurons, respectively. (D) 
Relationship between the pairwise-IG measure (oL' 1000) ) and the sum of 
connection weights (J^ + J21) under a 6-Hz oscillation. (E) Relationship 
between the single-IG measure (9fj 4, 1000) ) and the sum of connection 
weights Un + J21) under a 6-Hz oscillation. (F) Relationship between the 
single-IG measure (fl!j 4 ' 1000) ) a nd the sum of connection weights U12 
under a 40-Hz oscillation. 



J21) 



pairwise-IG measure can provide more direct information about 
the synaptic interactions between neurons than other correlation 
measures (Amari, 2009). It has been also shown that the pairwise- 
IG measure is statistically independent from the change in firing 
rate and that it provides pure neural interactions (Amari, 2001; 
Nakahara and Amari, 2002). Together with the findings in this 
study, the pairwise-IG measure is a very useful measure to study 
direct neural interactions between neurons. 

This study suggests that the actual values of the IG mea- 
sures depend on the mechanisms of oscillation. For an externally 

driven oscillation, 9^' ~ 0.2 was obtained for (/ 12 + /21) ~ 
l/N. For an internally induced oscillation, the same connection 
strength produced 0^ N ' ~ 0.002. Within the same oscillation 
mechanism, the selection of model neurons (binary model or 
Izhikevich model), or a small difference in network parame- 
ters such as phase differences also produced a difference in the 
actual value of the IG measures. Nonetheless, as long as the net- 
work is in one of the oscillation mechanisms, and the phase 



difference is kept the same, the IG measures can provide use- 
ful insights into network structures regardless of the oscillation 
frequencies. 

In the study of memory consolidation, one of the key ques- 
tions is to understand how the changes in synaptic connections 
are related to learning and memory formation. Evidence suggests 
that neural activity during slow-wave sleep plays an important 
role in learning (Diekelmann and Born, 2010). Specifically, there 
is increasing evidence supporting the hypothesis that replay of 
neural activity during subsequent sleep is positively correlated 
with memory formation (Pavlides and Winson, 1989; Wilson and 
McNaughton, 1994; Kudrimoti et al, 1999; Lee and Wilson, 2002; 
Euston et al, 2007; Girardeau et al, 2009; Peyrache et al, 2009b; 
Ego-Stengel and Wilson, 2010). However, the direct information 
about synaptic change is not available from multi-unit recordings 
of a freely behaving animal because spikes and local field poten- 
tials are the two main observables. In this study, we showed that 
0:f ' N ' was linearly related to the sum of the connection weights, 
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FIGURE 8 | Relationship between the IG measures and the sum of 
connection weights for an internally induced oscillation. Neuron 1 
was selected from the excitatory neuron pool and Neuron 2 was 
selected from the inhibitory neuron pool. In other words, one of the 
connections in + J21 was positive and the other was negative. (A) 
Average firing probability of 1250 Izhikevich neurons with approximately 
a 6-Hz oscillatory oscillation is shown. (B) Average firing probability of 
1250 Izhikevich neurons with approximately a 40-Hz oscillatory oscillation 
is shown. (C) Relationship between the pairwise-IG measure (9^' 1000) ) 



-2l— ■ ■ ■ ■ ■ • ■ ■ -J 
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J 12+J21 

and the sum of connection weights (J^ + Jn) under a 6-Hz oscillation. 
Black and gray lines represent the simulation results by binary neurons 
and Izhikevich neurons, respectively. (D) Relationship between the 
pairwise-IG measure (9^' 1000) ) and the sum of connection weights 
(J12 + J21) under a 6-Hz oscillation. (E) Relationship between the 
single-IG measure (9* 4 ' 1000) ) a nd the sum of connection weights 
U12 + J21) under a 6-Hz oscillation. (F) Relationship between the 
single-IG measure (9* 4, 1000) ) and the sum of connection weights 
(J12 + J21) under a 40-Hz oscillation. 



and that 9- ' was linearly related to the magnitude of external 
inputs, even under the oscillatory conditions. We have also ver- 
ified these relationships not only with a simple binary model 
neuron, but also with a more realistic spiking model neuron. This 
finding would allow us to analyze neural activity during slow- 
wave sleep before and after the task; 9^ 4, would be a good 

measure for the change of connection weight, and 9- N ' for 
the magnitude of background input that would be influenced 
by local field potentials. By comparing the relative change of 
9^ 4, N ^ between slow-wave sleeps before and after the task, and 
the strength of memory replay/improvement of behavior per- 
formance, the IG measure may provide a way to estimate the 
relationship between the synaptic modification and memory for- 
mation without having direct access to information of synaptic 
change. 

As a related approach to the IG method, the maximum entropy 
(MaxEnt) has attracted much attention recently (Schneidman 



et al, 2006; Tang et al, 2008; Tyler et al., 2012). The philoso- 
phy of the MaxEnt approach is not to assume anything other 
than what we know from the data. For example, if firing rate 
and pairwise correlation are the only information we have, the 
distribution with maximum entropy is given as the Boltzmann 
distribution, 

P (2) M = i exp fa fa + E ■ ( 14 ) 

where h' { is a bias term for the neuron i, ]'~ is the symmetric cou- 
pling strength between neurons i and and the partition function 
Z is given by, 

z = E ex p I E ^ + E4 x '*> ) • (15) 

fx) \ i i<J j 
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We see that the MaxEnt is equivalent to the IG with the 2nd-order 
LLM, 

iog/w .-.* = £ ef • % + £ e< 2 - - we)* N >, 

< »<J 

(16) 

where the relationship between the parameters are given as, 

9 (2,N) = ^ 9 e,N) =/ .. ;W (2.N) =logZ (17) 

As was discussed in Tatsuno and Okada (2004) and in Tatsuno 
et al. (2009), it is possible to relate these IG measures, 9- N ' and 
q(2, N)^ to networ k structure even for a network with asym- 
metric connections (Equation 5). However, under the influence 
of correlated inputs, we have also shown that the relationship 
in Equation 5 broke down, and that it was necessary to use the 
IG measures with the higher-order LLM such as the 4th-order 
(Equation 6) (Nie and Tatsuno, 2012). In other words, it was 
necessary to take into account neural activity of two additional 
neurons to estimate the direct neural interaction between neuron 
i and j. In summary, we see that the MaxEnt approach and the 
IG method are closely related. In addition, we also see that the 
MaxEnt can be considered a part of the IG method that provides 
a more general analysis framework for the space of the probability 
distributions. 

In this study, we used the synchronous neural activity for esti- 
mating the direct neural interaction as the form of (J y + /«•). 
However, in the real learning processes such as sequential learn- 
ing, it is possible that synaptic modification occurs differently 
for each direction; e.g., /y increases, while /j, decreases. The 
proposed method is not able to estimate the directed synaptic 
change. As one possible remedy for this difficulty, calculation of 
the pairwise-IG measure using the time-lagged spiking activity 
between neurons was suggested (Tatsuno and Okada, 2004; Nie 
and Tatsuno, 2012). Another limitation of the present study is 
not including the effect of delay; e.g., axonal conduction delay 
or synaptic transmission delay. It is possible that these delays 
dramatically change the firing patterns as well as increase a vari- 
ety of coexisting patterns (Izhikevich, 2006). Little is known 
about the relationship between the IG measures and direct neu- 
ral interactions with conduction delay. In addition, it has not 
been clear how IG measures with more neuronal interactions 
such as triplewise-IG measures 6j^' N ^ or quadruple-IG measures 

oL.;^ behave under oscillatory conditions. It would be inter- 
esting to extend the current study to include more neuronal 
interactions. 

Despite these limitations, the IG method is one of the most 
promising statistical tools for spike train analysis (Amari, 2001; 
Nakahara and Amari, 2002). Its direct relationship with the net- 
work parameters would provide useful information for the esti- 
mation of structural changes (Tatsuno and Okada, 2004; Tatsuno 
et al, 2009; Nie and Tatsuno, 2012). We hope that an advance- 
ment of novel analysis methods including IG will lead to a 
break-through finding in neuroscience. 
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